{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7c60ba5e",
   "metadata": {},
   "source": [
    "In this notebook, we demonstate is it possible to convert CNC to a Question Answering format. \n",
    "\n",
    "The questions can be either asking about the Cause or Effect, given the opposite argument. We created 6 simple templates to get the Question-Answers, and pick 2 templates per example. \n",
    "\n",
    "The final format of the data follows SQUAD columns (https://huggingface.co/datasets/squad_v2).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6b9c06f9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2257\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>corpus</th>\n",
       "      <th>doc_id</th>\n",
       "      <th>sent_id</th>\n",
       "      <th>eg_id</th>\n",
       "      <th>index</th>\n",
       "      <th>text</th>\n",
       "      <th>text_w_pairs</th>\n",
       "      <th>seq_label</th>\n",
       "      <th>pair_label</th>\n",
       "      <th>context</th>\n",
       "      <th>num_sents</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>cnc</td>\n",
       "      <td>train_01_0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>cnc_train_01_0_0_0</td>\n",
       "      <td>The State alleged they hacked Sabata Petros Ch...</td>\n",
       "      <td>The State alleged &lt;ARG1&gt;they hacked Sabata Pet...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>cnc</td>\n",
       "      <td>train_01_5</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>cnc_train_01_5_0_0</td>\n",
       "      <td>Police opened fire , killing 34 striking worke...</td>\n",
       "      <td>&lt;ARG1&gt;Police opened fire&lt;/ARG1&gt; , killing 34 s...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>cnc</td>\n",
       "      <td>train_01_5</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>cnc_train_01_5_0_1</td>\n",
       "      <td>Police opened fire , killing 34 striking worke...</td>\n",
       "      <td>&lt;ARG0&gt;Police opened fire&lt;/ARG0&gt; , &lt;ARG1&gt;&lt;SIG0&gt;...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>cnc</td>\n",
       "      <td>train_01_6</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>cnc_train_01_6_0_0</td>\n",
       "      <td>The three-member Farlam Commission , chaired b...</td>\n",
       "      <td>&lt;ARG1&gt;The three-member Farlam Commission , cha...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>cnc</td>\n",
       "      <td>train_01_6</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>cnc_train_01_6_0_1</td>\n",
       "      <td>The three-member Farlam Commission , chaired b...</td>\n",
       "      <td>The three-member Farlam Commission , chaired b...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  corpus      doc_id  sent_id  eg_id               index  \\\n",
       "0    cnc  train_01_0        0    0.0  cnc_train_01_0_0_0   \n",
       "1    cnc  train_01_5        0    0.0  cnc_train_01_5_0_0   \n",
       "2    cnc  train_01_5        0    1.0  cnc_train_01_5_0_1   \n",
       "3    cnc  train_01_6        0    0.0  cnc_train_01_6_0_0   \n",
       "4    cnc  train_01_6        0    1.0  cnc_train_01_6_0_1   \n",
       "\n",
       "                                                text  \\\n",
       "0  The State alleged they hacked Sabata Petros Ch...   \n",
       "1  Police opened fire , killing 34 striking worke...   \n",
       "2  Police opened fire , killing 34 striking worke...   \n",
       "3  The three-member Farlam Commission , chaired b...   \n",
       "4  The three-member Farlam Commission , chaired b...   \n",
       "\n",
       "                                        text_w_pairs  seq_label  pair_label  \\\n",
       "0  The State alleged <ARG1>they hacked Sabata Pet...          1           1   \n",
       "1  <ARG1>Police opened fire</ARG1> , killing 34 s...          1           1   \n",
       "2  <ARG0>Police opened fire</ARG0> , <ARG1><SIG0>...          1           1   \n",
       "3  <ARG1>The three-member Farlam Commission , cha...          1           1   \n",
       "4  The three-member Farlam Commission , chaired b...          1           1   \n",
       "\n",
       "   context  num_sents  \n",
       "0      NaN          1  \n",
       "1      NaN          1  \n",
       "2      NaN          1  \n",
       "3      NaN          1  \n",
       "4      NaN          1  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import re\n",
    "import json\n",
    "import numpy as np\n",
    "np.random.seed(42)\n",
    "\n",
    "\n",
    "data = pd.read_csv(\"../data/V2/train_subtask2.csv\")\n",
    "print(len(data))\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9a438a0f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('What resulted from \"the allocation of low cost ( RDP ) houses at Marikana West Extension 2\"?', {'text': 'they hacked Sabata Petros Chale , 39 , to death in Marikana West , on December 8 , 2016'})\n",
      "('What happened because of \"the allocation of low cost ( RDP ) houses at Marikana West Extension 2\"?', {'text': 'they hacked Sabata Petros Chale , 39 , to death in Marikana West , on December 8 , 2016'})\n"
     ]
    }
   ],
   "source": [
    "def get_cause_effect_spans(text_w_pairs):\n",
    "    cause=re.sub(r\"\\<.*?\\>\",\"\",re.search(r\"\\<ARG0\\>(.*?)\\<\\/ARG0\\>\",text_w_pairs).group(1))\n",
    "    effect=re.sub(r\"\\<.*?\\>\",\"\",re.search(r\"\\<ARG1\\>(.*?)\\<\\/ARG1\\>\",text_w_pairs).group(1))\n",
    "    return cause, effect\n",
    "\n",
    "# Non-Extractive QA --> Not relevant to SQUAD, but might be useful for other cases\n",
    "def seqtemplate1(seq_label):\n",
    "    question='Is this sentence Causal or Non-causal?'\n",
    "    answers={\"text\": [\"Causal\" if int(seq_label)==1 else \"Non-causal\"]}\n",
    "    return question, answers\n",
    "\n",
    "def seqtemplate2(seq_label):\n",
    "    question='Are there any causal relations in this sentence?'\n",
    "    answers={\"text\": [\"Yes\" if int(seq_label)==1 else \"No\"]}\n",
    "    return question, answers\n",
    "\n",
    "def spantemplate1(text_w_pairs):\n",
    "    question=f'Did \"{cause}\" lead to \"{effect}\"?'\n",
    "    answers={\"text\": \"Yes\"}\n",
    "    return question, answers \n",
    "\n",
    "def spantemplate2(text_w_pairs):\n",
    "    cause, effect = get_cause_effect_spans(text_w_pairs)\n",
    "    question=f'Did \"{effect}\" lead to \"{cause}\"?'\n",
    "    answers={\"text\": \"No\"}\n",
    "    return question, answers \n",
    "\n",
    "# Extractive QA\n",
    "def spantemplate3(text_w_pairs):\n",
    "    cause, effect = get_cause_effect_spans(text_w_pairs)\n",
    "    question=f'What caused \"{effect}\"?'\n",
    "    answers={\"text\": cause}\n",
    "    return question, answers \n",
    "\n",
    "def spantemplate4(text_w_pairs):\n",
    "    cause, effect = get_cause_effect_spans(text_w_pairs)\n",
    "    question=f'What led to \"{effect}\"?'\n",
    "    answers={\"text\": cause}\n",
    "    return question, answers \n",
    "\n",
    "def spantemplate5(text_w_pairs):\n",
    "    cause, effect = get_cause_effect_spans(text_w_pairs)\n",
    "    question=f'Why did \"{effect}\" occur?'\n",
    "    answers={\"text\": cause}\n",
    "    return question, answers \n",
    "\n",
    "def spantemplate6(text_w_pairs):\n",
    "    cause, effect = get_cause_effect_spans(text_w_pairs)\n",
    "    question=f'What resulted from \"{cause}\"?'\n",
    "    answers={\"text\": effect}\n",
    "    return question, answers \n",
    "\n",
    "def spantemplate7(text_w_pairs):\n",
    "    cause, effect = get_cause_effect_spans(text_w_pairs)\n",
    "    question=f'What happened because of \"{cause}\"?'\n",
    "    answers={\"text\": effect}\n",
    "    return question, answers \n",
    "\n",
    "def spantemplate8(text_w_pairs):\n",
    "    cause, effect = get_cause_effect_spans(text_w_pairs)\n",
    "    question=f'What did \"{cause}\" cause?'\n",
    "    answers={\"text\": effect}\n",
    "    return question, answers \n",
    "\n",
    "\n",
    "# Show example conversions\n",
    "text_w_pairs = data['text_w_pairs'][0]\n",
    "extractiveQA_templates = [spantemplate3, spantemplate4, spantemplate5, spantemplate6, spantemplate7, spantemplate8]\n",
    "templates = np.random.choice(extractiveQA_templates, size=2)\n",
    "for template in templates:\n",
    "    print(template(text_w_pairs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "321e7146",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title</th>\n",
       "      <th>context</th>\n",
       "      <th>question</th>\n",
       "      <th>answers</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>cnc_train_01_0_0_0_0</td>\n",
       "      <td>cnc</td>\n",
       "      <td>The State alleged they hacked Sabata Petros Ch...</td>\n",
       "      <td>Why did \"they hacked Sabata Petros Chale , 39 ...</td>\n",
       "      <td>{'text': 'the allocation of low cost ( RDP ) h...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>cnc_train_01_0_0_0_1</td>\n",
       "      <td>cnc</td>\n",
       "      <td>The State alleged they hacked Sabata Petros Ch...</td>\n",
       "      <td>What happened because of \"the allocation of lo...</td>\n",
       "      <td>{'text': 'they hacked Sabata Petros Chale , 39...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>cnc_train_01_5_0_0_0</td>\n",
       "      <td>cnc</td>\n",
       "      <td>Police opened fire , killing 34 striking worke...</td>\n",
       "      <td>What happened because of \"trying to disperse a...</td>\n",
       "      <td>{'text': 'Police opened fire'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>cnc_train_01_5_0_0_1</td>\n",
       "      <td>cnc</td>\n",
       "      <td>Police opened fire , killing 34 striking worke...</td>\n",
       "      <td>What led to \"Police opened fire\"?</td>\n",
       "      <td>{'text': 'trying to disperse a group gathered ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>cnc_train_01_5_0_1_0</td>\n",
       "      <td>cnc</td>\n",
       "      <td>Police opened fire , killing 34 striking worke...</td>\n",
       "      <td>Why did \"killing 34 striking workers and wound...</td>\n",
       "      <td>{'text': 'Police opened fire'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4509</th>\n",
       "      <td>cnc_train_09_B_160_0_0_1</td>\n",
       "      <td>cnc</td>\n",
       "      <td>Mullanchira Mathai was beaten to death on Janu...</td>\n",
       "      <td>Why did \"the acquittal of 11 CPI(M) activists ...</td>\n",
       "      <td>{'text': 'lack of witnesses'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4510</th>\n",
       "      <td>cnc_train_09_A_161_0_0_0</td>\n",
       "      <td>cnc</td>\n",
       "      <td>Post offices wore a deserted look as members o...</td>\n",
       "      <td>Why did \"Post offices wore a deserted look\" oc...</td>\n",
       "      <td>{'text': 'members of the National Federation o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4511</th>\n",
       "      <td>cnc_train_09_A_161_0_0_1</td>\n",
       "      <td>cnc</td>\n",
       "      <td>Post offices wore a deserted look as members o...</td>\n",
       "      <td>What resulted from \"members of the National Fe...</td>\n",
       "      <td>{'text': 'Post offices wore a deserted look'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4512</th>\n",
       "      <td>cnc_train_09_B_161_0_0_0</td>\n",
       "      <td>cnc</td>\n",
       "      <td>Muttukad Nanappan was stabbed to death on June...</td>\n",
       "      <td>Why did \"this case too did not stand\" occur?</td>\n",
       "      <td>{'text': 'lack of witnesses'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4513</th>\n",
       "      <td>cnc_train_09_B_161_0_0_1</td>\n",
       "      <td>cnc</td>\n",
       "      <td>Muttukad Nanappan was stabbed to death on June...</td>\n",
       "      <td>What did \"lack of witnesses\" cause?</td>\n",
       "      <td>{'text': 'this case too did not stand'}</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4514 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                            id title  \\\n",
       "0         cnc_train_01_0_0_0_0   cnc   \n",
       "1         cnc_train_01_0_0_0_1   cnc   \n",
       "2         cnc_train_01_5_0_0_0   cnc   \n",
       "3         cnc_train_01_5_0_0_1   cnc   \n",
       "4         cnc_train_01_5_0_1_0   cnc   \n",
       "...                        ...   ...   \n",
       "4509  cnc_train_09_B_160_0_0_1   cnc   \n",
       "4510  cnc_train_09_A_161_0_0_0   cnc   \n",
       "4511  cnc_train_09_A_161_0_0_1   cnc   \n",
       "4512  cnc_train_09_B_161_0_0_0   cnc   \n",
       "4513  cnc_train_09_B_161_0_0_1   cnc   \n",
       "\n",
       "                                                context  \\\n",
       "0     The State alleged they hacked Sabata Petros Ch...   \n",
       "1     The State alleged they hacked Sabata Petros Ch...   \n",
       "2     Police opened fire , killing 34 striking worke...   \n",
       "3     Police opened fire , killing 34 striking worke...   \n",
       "4     Police opened fire , killing 34 striking worke...   \n",
       "...                                                 ...   \n",
       "4509  Mullanchira Mathai was beaten to death on Janu...   \n",
       "4510  Post offices wore a deserted look as members o...   \n",
       "4511  Post offices wore a deserted look as members o...   \n",
       "4512  Muttukad Nanappan was stabbed to death on June...   \n",
       "4513  Muttukad Nanappan was stabbed to death on June...   \n",
       "\n",
       "                                               question  \\\n",
       "0     Why did \"they hacked Sabata Petros Chale , 39 ...   \n",
       "1     What happened because of \"the allocation of lo...   \n",
       "2     What happened because of \"trying to disperse a...   \n",
       "3                     What led to \"Police opened fire\"?   \n",
       "4     Why did \"killing 34 striking workers and wound...   \n",
       "...                                                 ...   \n",
       "4509  Why did \"the acquittal of 11 CPI(M) activists ...   \n",
       "4510  Why did \"Post offices wore a deserted look\" oc...   \n",
       "4511  What resulted from \"members of the National Fe...   \n",
       "4512       Why did \"this case too did not stand\" occur?   \n",
       "4513                What did \"lack of witnesses\" cause?   \n",
       "\n",
       "                                                answers  \n",
       "0     {'text': 'the allocation of low cost ( RDP ) h...  \n",
       "1     {'text': 'they hacked Sabata Petros Chale , 39...  \n",
       "2                        {'text': 'Police opened fire'}  \n",
       "3     {'text': 'trying to disperse a group gathered ...  \n",
       "4                        {'text': 'Police opened fire'}  \n",
       "...                                                 ...  \n",
       "4509                      {'text': 'lack of witnesses'}  \n",
       "4510  {'text': 'members of the National Federation o...  \n",
       "4511      {'text': 'Post offices wore a deserted look'}  \n",
       "4512                      {'text': 'lack of witnesses'}  \n",
       "4513            {'text': 'this case too did not stand'}  \n",
       "\n",
       "[4514 rows x 5 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Run for whole dataset\n",
    "\n",
    "qa_data = []\n",
    "qa_data_columns = ['id','title','context','question','answers']\n",
    "for i,row in data.iterrows():\n",
    "    templates = np.random.choice(extractiveQA_templates, size=2)\n",
    "    for j,template in enumerate(templates):\n",
    "        question, answers = template(row['text_w_pairs'])\n",
    "        unique_index = row['index']\n",
    "        qa_data.append({\n",
    "            'id':f'{unique_index}_{j}',\n",
    "            'title':row.corpus,\n",
    "            'context':row.text,\n",
    "            'question':question,\n",
    "            'answers':answers\n",
    "        })\n",
    "\n",
    "pd.DataFrame(qa_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1ab8b6cf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': 'cnc_train_01_0_0_0_0',\n",
       " 'title': 'cnc',\n",
       " 'context': 'The State alleged they hacked Sabata Petros Chale , 39 , to death in Marikana West , on December 8 , 2016 , allegedly over the allocation of low cost ( RDP ) houses at Marikana West Extension 2 .',\n",
       " 'question': 'Why did \"they hacked Sabata Petros Chale , 39 , to death in Marikana West , on December 8 , 2016\" occur?',\n",
       " 'answers': {'text': 'the allocation of low cost ( RDP ) houses at Marikana West Extension 2'}}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qa_data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "79d2d0ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "# qa_data.to_csv(\"../data/V2/cnc_train_squad.csv\", index=False, encoding='utf-8-sig')\n",
    "with open(\"../data/V2/cnc_train_squad.json\", 'w') as f:\n",
    "    json.dump(qa_data, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f85e1bf",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3127e88f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
